Search for: All records

Standard training for Multi-modal Large Language Models (MLLMs) involves concatenating non-textual information, like vision or audio, with a text prompt. This approach may not encourage deep integration of modalities, limiting the model's ability to leverage the core language model's reasoning capabilities. This work examined the impact of interleaved instruction tuning in an audio MLLM, where audio tokens are interleaved within the prompt. Using the Listen, Think, and Understand (LTU) model as a testbed, we conduct an experiment using the Synonym and Hypernym Audio Reasoning Dataset (SHARD), our newly created reasoning benchmark for audio-based semantic reasoning focusing on synonym and hypernym recognition. Our findings show that while even zero-shot interleaved prompting improves performance on our reasoning tasks, a small amount of fine-tuning using interleaved training prompts improves the results further, however, at the expense of the MLLM's audio labeling ability. 

To address the challenge of short-term object pose tracking in dynamic environments with monocular RGB input, we introduce a large-scale synthetic dataset Omni-Pose6D, crafted to mirror the diversity of real-world conditions. We additionally present a benchmarking framework for a comprehensive comparison of pose tracking algorithms. We propose a pipeline featuring an uncertainty-aware keypoint refinement network, employing probabilistic modeling to refine pose estimation. Comparative evaluations demonstrate that our approach achieves performance superior to existing baselines on real datasets, underscoring the effectiveness of our synthetic dataset and refinement technique in enhancing tracking precision in dynamic contexts. Our contributions set a new precedent for the development and assessment of object pose tracking methodologies in complex scenes. 

Abstract Ferrimagnetic oxide thin films are important material platforms for spintronic devices. Films grown on low symmetry orientations such as (110) exhibit complex anisotropy landscapes that can provide insight into novel phenomena such as spin‐torque auto‐oscillation and spin superfluidity. Using spin‐Hall magnetoresistance measurements, the in‐plane (IP) and out‐of‐plane (OOP) uniaxial anisotropy energies are determined for a thickness series (5–50 nm) of europium iron garnet (EuIG) and thulium iron garnet (TmIG) films epitaxially grown on a gadolinium gallium substrate with (110) orientation and capped with Pt. Pt/EuIG/GGG exhibits an (001) easy plane of magnetization perpendicular to the substrate, whereas Pt/TmIG/GGG exhibits an (001) hard plane of magnetization perpendicular to the substrate with an IP easy axis. Both IP and OOP surface anisotropy energies comparable in magnitude to the bulk anisotropy are observed. The temperature dependence of the surface anisotropies is consistent with first‐order predictions of a simplified Néel surface anisotropy model. By taking advantage of the thickness and temperature dependence demonstrated in these ferrimagnetic oxides grown on the low symmetry (110) orientations, the complex anisotropy landscapes can be tuned to act as a platform to explore rich spin textures and dynamics.